Benchmarking and Resource Measurement
نویسندگان
چکیده
Proper benchmarking and resource measurement is an important topic, because benchmarking is a widely-used method for the comparative evaluation of tools and algorithms in many research areas. It is essential for researchers, tool developers, and users, as well as for competitions. We formulate a set of requirements that are indispensable for reproducible benchmarking and reliable resource measurement of automatic solvers, verifiers, and similar tools, and discuss limitations of existing methods and benchmarking tools. Fulfilling these requirements in a benchmarking framework is complex and can (on Linux) currently only be done by using the cgroups feature of the kernel. We provide BenchExec, a ready-to-use, tool-independent, and free implementation of a benchmarking framework that fulfills all presented requirements, making reproducible benchmarking and reliable resource measurement easy. Our framework is able to work with a wide range of different tools and has proven its reliability and usefulness in the International Competition on Software Verification.
منابع مشابه
An Efficiency Measurement and Benchmarking Model Based on Tobit Regression, GANN-DEA and PSOGA
The purpose of this study is designing a model based on Tobit regression, DEA, Artificial Neural Network, Genetic Algorithm and Particle Swarm Optimization to evaluate the efficiency and also benchmarking the efficient and inefficient units. This model has three stages, and it uses the data envelopment analysis combined model with neural network, optimized by genetic algorithm, to evaluate the ...
متن کاملPractical benchmarking in DEA using artificial DMUs
Data envelopment analysis (DEA) is one of the most efficient tools for efficiency measurement which can be employed as a benchmarking method with multiple inputs and outputs. However, DEA does not provide any suggestions for improving efficient units, nor does it provide any benchmark or reference point for these efficient units. Impracticability of these benchmarks under environmental conditio...
متن کاملValue for money - Nowhere else on the web will you find access to the depth or range of information at such a low cost - Indiv
Describes the aims, reasons for development, key features, and uniqueness of an on-line Benchmarking and Performance Improvement Resource. The paper begins by describing the historical context within which the website has been developed. It describes the growth in the use of business excellence, benchmarking and performance measurement, and how these improvement approaches are related to each o...
متن کاملارزشیابی فناوری اطلاعات با رویکردی تلفیقی
The IT value measurement model proposed to evaluate the business value of IT. This model utilizes some different tools and techniques such as: Benchmarking, Balanced Scorecard, qualitative and quantitative measurement techniques. The basis of this model is a link between 3-layer IT classification and IT planning with a combinatorial approach. Connecting these three layers with effectiveness an...
متن کاملAn integrated data envelopment analysis–artificial neural network approach for benchmarking of bank branches
Efficiency and quality of services are crucial to today’s banking industries. The competition in this section has become increasingly intense, as a result of fast improvements in Technology. Therefore, performance analysis of the banking sectors attracts more attention these days. Even though data envelopment analysis (DEA) is a pioneer approach in the literature as of an efficiency measurement...
متن کامل